Overview

NAVO has started regularly querying some TAP and Cone Search services to collect data on their response times. So far this is mostly NAVO services, but also includes a CDS 2MASS cone search for comparison. (Some Chandra Source Catalog queries are also done, but due to sparse sky coverage these need to be adjusted.)

The queries are done using the servicemon application (https://servicemon.readthedocs.io/en/latest/), and are executed from several different locations. The AWS instrumentation is handled with the software at https://github.com/NASA-NAVO/AWS_servicemon. The results are written to a TAP-accessible database currently running at IPAC.

Colaborating

Now that all can examine the monitoring data and run additional tests, all can contribute:

• Analyzing response data.

  • Explaining anomalous or poor measurements (is the issue with the service, network, servicemon, etc.?)
  • Addressing major performance issues.

• Developing other plots, analysis or alerts.

  • What plots (like those below?) are worth posting somewhere on web?

• Maintaining the operational monitoring.

• Monitoring parameters

  • Should services be added/removed from the monitoring list?
  • Is the current cadence OK?
  • Do we agree that we should add in non-random queries so we have more concrete comparison data?
  • TAP queries are currently all async. We've found this adds significant overhead in some cases. Should routine or one off monitoring include sync TAP queries?

• Open Development participation in servicemon and AWS_servicemon.
  • Filing, discussing and prioritizing issues.
  • Bug fix and feature development.

Known Issues/Action Items

Short term:

• We're not capturing all the int(n) columns in the DB. Tap results have values for 2 < n <= 7 that we're not storing.
• int(n) columns are inconsistent between tap and cone. It would be nicer if the common columns came first, then tap-specific ones came after.
• The base_name values are not consistent. If we agree, they should probably follow the convention of the form \\_. E.g., _IPAC2MASS, _STScIObsTAP, _HEASARCswiftmastr.|
• Probably should add some non-random queries to suite.
• Ensure start_time and end_time columns are indexed.

Longer term:

• Consider controlling inputs on a per service basis due to differences in density and coverage.
• Support non-positional queries (TAP, DataLink, non-VO services)

What Tests Are Run

All of the parameters of the queries are configurable, but below is what is currently running. TAP queries now are all async.

Services

base_name	service_type
2MASS_STScI	cone
CDS_2MASS	cone
CSC	cone
CSC	tap
HEASARC_swiftmastr	cone
HEASARC_swiftmastr	tap
HEASARC_xmmssc	cone
HEASARC_xmmssc	tap
IPAC_2MASS	cone
IPAC_2MASS	tap
IPAC_WISE	cone
IPAC_WISE	tap
NED	cone
NED	tap
PanSTARRS	tap
PanSTARRS	xcone
STScI_ObsTAP	tap
WISE_ST	cone

When and what cones?

A set of 10 random cone queries, with radii ranging from 0 to 0.25 degrees, is run for each service every 6 hours. The exact hours are staggered by location.

We should change this to include (or only use) fixed cones, so that we can compare the exact same queries over time. (servicemon can be run with fixed or random targets.)

From Where

The queries are run from the following AWS regions:

'ap-northeast-1', 'ap-southeast-2', 'eu-west-3', 'sa-east-1', 'us-east-1','us-west-2'

Due to testing, the database also contains results from other locations like:

'ip-172-31-36-250.us-west-2.compute.internal', 'ip-172-31-43-179.ap-northeast-1.compute.internal', 'kvmexodev.ipac.caltech.edu'

Result Data Available via TAP

The TAP service at http://navo01.ipac.caltech.edu/TAP has a table called navostats with one row per query run by servicemon.

Note: The VOSI endpoints have not yet been implemented for this service, so PyVO and Topcat will complain during metadata gathering, but both both PyVO and Topcat can be used to query this service, and all the TAP_SCHEMA tables are implemented, so those can be used to query metadata.

The following columns are available:

Query Description

Query Input

column_name	datatype	format	description
ra	double	20.6f	Right Ascension of the query cone region.
dec	double	20.6f	Declination of the query cone region.
sr	double	20.6f	Radius of the query cone region (deg).
adql	char	300s	For TAP queries this is the full ADQL query that was done. Empty for non-TAP queries.

Other Query Metadata

column_name	datatype	format	description
access_url	char 300s	The base URL of the service.
base_name	char	20s	A short name of the service given by the servicemon configuration files. Not yet consistent for all services.
service_type	char	20s	While other values are possible, the main service types we're tracking now are tap, cone, and xcone which is like cone, but not VO-compliant.
location	char	80s	Self-declared location of the monitoring service (e.g., AWS region).
start_time	char	30s	The data and time that the query was started (format='%Y-%m-%d %H:%M:%S.%f').
end_time	char	30s	The data and time that the query was completed (format='%Y-%m-%d %H:%M:%S.%f').

Query Results

Note that these values may empty for certain types of query failures.

Timing

column_name	datatype	format	description
int0_desc	char	20s	Description of the first interval measured (int0).
int0_duration	double	20.6f	The duration of the first interval measured. So far this is the time to an HTTP response indicating that the query is complete, but prior to the results being streamed back to the client.
int1_desc	char	20s	Description of the second interval measured (int1).
int1_duration	double	20.6f	The duration of the second interval measured. So far this is the time to download the the results after the HTTP response indicating that the query was complete.
int2_desc	char	20s	Description of the third interval measured (int1=2).
int1=2_duration	double	20.6f	For cone and sync tap queries, this is the total duration from when the query was issued until result download was complete. For async TAP, this is the amount of time it took to submit the async query.

Result metadata

column_name	datatype	format	description
num_columns	integer	9d	Number of FIELDs in the result VOTable.
num_rows	integer	9d	Number of rows in the result VOTable.
size	integer	10d	Size of the result VOTable (bytes).

Querying and Plotting the Data

Imports

This code requires an environment that includes servicemon, bokeh and pandas.

from servicemon.analysis.stat_queries import StatQueries

from bokeh.plotting import figure, output_file, output_notebook, show, reset_output
from bokeh.models import Legend, ColumnDataSource, HoverTool, Div
from bokeh.transform import factor_cmap, factor_mark
from bokeh.layouts import layout

import pandas as pd

Sample Plotting Functions

These functions support converting our query results to pandas then plotting some sample plots using bokeh, both in a notebook and on a web page.

Create Data Source for Bokeh

def create_source(data):
    """
    Create a Bokeh ColumnDataSource from an Astropy Table containing results from
    a navostats query.
    
    
    Parameters
    ----------
    data : astropy.table.Table
        A table presumed to contain the results from a query on navostats.
        In particular, the following columns must be present: location, start_time, 
        int0_duration, int1_duration, num_rows, base_name, service_type, ra, dec, sr

    Returns
    -------
    ColumnDataSource
        A Bokeh data source suitable for plotting
    """
    # Masked values in integer columns show up as <NA> when exported to Pandas.
    # Such values seem to cause weird errors when displayed in bokeh, even when those rows
    # are filtered out with dropna().  So convert the column to float, which results
    # in masked values being NaN (np.nan) which are smoothly ignored by bokeh.
    data["num_rows"] = data["num_rows"].astype(float)
    
    # Create the Pandas DataFrame, with start time officially being a datetime.
    df = data["location", "start_time", "int0_duration", "int1_duration", "num_rows", "base_name", "service_type",
              "ra", "dec", "sr"].to_pandas().copy()
    df["dt_start_time"] = pd.to_datetime(df["start_time"], format='%Y-%m-%d %H:%M:%S.%f')
    
    # Create the bokeh data source from the data frame.
    source = ColumnDataSource(df)
    
    return source

Support tooltips for plotted points.

def create_hover():
    """
    Returns
    -------
    HoverTool
        A Bokeh hover tool which can provide a tooltip for plotted data points.
    """
    hover = HoverTool(
        tooltips=[
            ("Cone", "@ra, @dec, @sr"),
            ("Query Time", "@int0_duration"),
            ("Download Time", "@int1_duration"),
            ("# of Rows", "@num_rows"),
            ("Location", "@location"),
            ("Start Time", "@dt_start_time{%m/%d %H:%M:%S}")
        ],
        formatters={
            '@dt_start_time': 'datetime'
        })
    return hover

Plot Query Duration versus Start Time

def create_plot_dur_v_start_time(source, y_axis_type='log', y_range=(0.001, 10**3)):
    """
    Create a Bokeh plot (Figure) of int0_duration versus start_time.
    
    Parameters
    ----------
    source : ColumnDataSource
        Bokeh data source containing the navostats data
    y_axis_type : str
        auto, linear, log, datetime, or mercator
    y_range : tuple (min, max)
        The range of values to display on the y axis.  When y_axis_type is 'log',
        it helps if the endpoints are exact powers of 10.

    Returns
    -------
    Figure
        A Bokeh plot that can be shown.
    """
    
    # create plot with a datetime axis type
    p = figure(plot_width=700, plot_height=500, x_axis_type="datetime", 
               y_axis_type=y_axis_type, y_range=y_range)
    
    hover = create_hover()
    p.add_tools(hover)

    # add renderers
    qt_rend = p.circle(x="dt_start_time", y="int0_duration", source=source, size=4, color='red', alpha=0.2)

    p.title.text = "Query Duration v. Start Time"
    p.xaxis.axis_label = 'Start Time'
    p.yaxis.axis_label = 'Duration (s)'

    return p

Plot Query and Download Duration versus Number of Result Rows

def create_plot_durations_v_nrows(source, x_axis_type='log', x_range=(1, 10**5), 
                             y_axis_type='log', y_range=(0.001, 10**3)):
    """
    Create a Bokeh plot (Figure) of int0_duration and int1_duration versus num_rows.
    num_rows is the number of result rows from the query.
    
    Parameters
    ----------
    source : ColumnDataSource
        Bokeh data source containing the navostats data
    x_axis_type : str
        auto, linear, log, datetime, or mercator
    x_range : tuple (min, max)
        The range of values to display on the x axis.  When x_axis_type is 'log',
        it helps if the endpoints are exact powers of 10.
    y_axis_type : str
        auto, linear, log, datetime, or mercator
    y_range : tuple (min, max)
        The range of values to display on the y axis.  When y_axis_type is 'log',
        it helps if the endpoints are exact powers of 10.

    Returns
    -------
    Figure
        A Bokeh plot that can be shown.
    """
    # create a new plot with a datetime axis type
    p = figure(plot_width=500, plot_height=500, 
               x_axis_type=x_axis_type, x_range=x_range,
               y_axis_type=y_axis_type, y_range=y_range)
    
    hover = create_hover()
    p.add_tools(hover)

    # add renderers
    qt_rend = p.circle(x="num_rows", y="int0_duration", source=source, size=4, color='red', alpha=0.2)
    dt_rend = p.circle(x="num_rows", y="int1_duration", source=source, size=4, color='green', alpha=0.2)

    legend = Legend(items=[
        ("Query Duration",   [qt_rend]),
        ("Download Duration", [dt_rend])
    ], location=(0, 40), click_policy='hide')
    p.add_layout(legend, 'below')

    p.title.text = 'Query and Download Durations v. # of Rows'
    p.xaxis.axis_label = '# of Rows'
    p.yaxis.axis_label = 'Durations (s)'
    
    return p 

Create Plots for all Services

Loop through stats for multiple services and display the plots in the notebook or on an html page.

def create_service_plots(stat_queries, services, start_time=None, end_time=None, htmlfile=None, title=None):
    """    Create a Bokeh plot (Figure) of int0_duration and int1_duration versus num_rows.
    num_rows is the number of result rows from the query.
    
    Parameters
    ----------
    stat_queries : servicemon.analysis.stat_queriesStatQueries
        A StatQueries object which will perform the navostats queries.
    services : List of tuples
        Each tuple should be a doubleton indicating a (base_name, service_type)
        that should be queried and plotted.
    start_time : str
        The beginning of a time window bounding the query.  Format is '%Y-%m-%d %H:%M:%S.%f'.
        Least significant part can be omitted as the comparisons done are just alphabetic.
    end_time : str
        The end of a time window bounding the query.  Format is '%Y-%m-%d %H:%M:%S.%f'.
        Least significant part can be omitted as the comparisons done are just alphabetic.
    htmlfile : str
        The name of the output html file.  Specify None if notebook output is desired.
    title : str
        The title given to the output html page.  Ignored if htmlfile is None.

    Returns
    -------
    None
    """
    reset_output()
    if htmlfile is None:
        output_notebook()
    else:
        output_file(htmlfile, title=title)
        
    layout_children = []
    
    for s in services:
        base_name = s[0]
        service_type = s[1]
        row_title = f'{base_name} {service_type}'
        
        data = stat_queries.do_stat_query(base_name=base_name, service_type=service_type, start_time=start_time, end_time=end_time)
        source = create_source(data)
        
        over_time_plot = create_plot_dur_v_start_time(source)
        time_v_nrows_plot = create_plot_durations_v_nrows(source)
        
        layout_children.append([Div(text=f'<h1>{row_title}</h1>')])
        layout_children.append([over_time_plot, time_v_nrows_plot])
    
    l = layout(children=layout_children)
    show(l)

Example: Use Above Methods

sq = StatQueries()

services = sq.get_name_service_pairs()

create_service_plots(sq, services, start_time='2021-02-02')
#create_service_plots(sq, services[0:2], start_time='2021-02-10', htmlfile='testpage.html', title="Test Title")

Loading BokehJS ...

More Plot Ideas

• Differentiate based on where the query originated. E.g., plot different shape or color based on the location value.
• Plot durations versus time of day, overlaying multiple days, to look for trends based on time of day. Since the queries are started at different hours depending on the location, plotting the locations with different colors/shapes as above could also be helpful.

Sample plot differentiating location by shape

def create_plot_location_shapes(ds):

    LOCATIONS = ['ap-northeast-1',
                 'ap-southeast-2',
                 'eu-west-3',
                 'sa-east-1',
                 'us-east-1',
                 'us-west-2'
                ]
                 
    MARKERS = ['asterisk', 
               'circle',
               'diamond',
               'plus',
               'square',
               'triangle'
              ]
    
    p = figure(plot_width=500, plot_height=500, 
               x_axis_type='log', x_range=(0.001, 10**3),
               y_axis_type='log', y_range=(1, 10**5))
    p.title.text = 'Duration v. # of Rows'
    p.xaxis.axis_label = 'Duration (s)'
    p.yaxis.axis_label = '# of Rows'
    
    s = p.scatter("int0_duration", "num_rows", source=source, 
              legend_field="location", 
              fill_alpha=0.4, size=4,
              marker=factor_mark('location', MARKERS, LOCATIONS),
              color='red', 
              alpha=0.2)
    
    # Commenting this out for now.  Drawing the location shapes this way exposes a limitation
    # with Bokeh where clicking anywhere in the legend hides everything.
    # I think the locations would have to be drawn separately for the legend click hiding to 
    # work as expected.
    #p.legend.click_policy="hide"
    
    p.legend.location=(0, 0)
    p.add_layout(p.legend[0], 'below')
    
    return p

Try out the location plots

reset_output()
output_notebook()

query = """
select * from navostats
where location in (
   'ap-northeast-1',
   'ap-southeast-2',
   'eu-west-3',
   'sa-east-1',
   'us-east-1',
   'us-west-2'
)
"""
data = sq.do_query(query)
source = create_source(data)

plot = create_plot_location_shapes(source)
show(plot)

Loading BokehJS ...